Surround ambient light sensing, processing and adjustment

ABSTRACT

Directional image sensor data may be acquired with one or more directional image sensors. A light source and illumination image may be generated based on the directional image sensor data. A number of operations may be caused to be performed for an image based at least in part on light source information in the light source image. The operations may include display management operations, device positional operations, augmented reality superimposition operations, ambient light control operations, etc.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present invention claims the benefit of U.S. Provisional PatentApplication No. 62/195,107, filed on 21 Jul. 2015, which is herebyincorporated by reference.

TECHNOLOGY

The present invention relates generally to vision devices, and inparticular, to surround ambient light sensing, processing and adjustmentfor vision devices.

BACKGROUND

Video materials may be created and color graded with specific artisticintent in professional production settings. These professionalproduction settings may be very different from viewing environments inwhich the video materials are to be rendered. As a result, instead ofseeing vivid, highly detailed images with intended color properties,viewers may see dull, washed out, poorly colored images thatsignificantly deviate from the artistic intent with which the videomaterials were initially created and color graded in the professionalproduction settings.

Vision devices such as consumer televisions, mobile devices, etc., mayattempt to adjust picture mapping to compensate for ambient light basedon measurements of global ambient light levels. However, such adjustmentand compensation (e.g., non-directional, brightness only, etc.) may betoo simplistic to generate full precise real time picture mapping foreffectively alleviating ambient light problems.

The approaches described in this section are approaches that could bepursued, but not necessarily approaches that have been previouslyconceived or pursued. Therefore, unless otherwise indicated, it shouldnot be assumed that any of the approaches described in this sectionqualify as prior art merely by virtue of their inclusion in thissection. Similarly, issues identified with respect to one or moreapproaches should not assume to have been recognized in any prior art onthe basis of this section, unless otherwise indicated.

BRIEF DESCRIPTION OF DRAWINGS

The present invention is illustrated by way of example, and not by wayof limitation, in the figures of the accompanying drawings and in whichlike reference numerals refer to similar elements and in which:

FIG. 1A illustrates an example light map based controller comprising alight source identification module operating in conjunction withdirectional image sensors;

FIG. 1B illustrates an example operational environment in which thelight map based controller may operate;

FIG. 1C illustrates an example light source image;

FIG. 2 depicts a block diagram of an example light map based controller;

FIG. 3A illustrates an example configuration in which a display manageruses light map based control output to perform display managementoperations in an image rendering environment;

FIG. 3B illustrates an example configuration in which a display managerand a device location manager use light map based control output toperform display management operations, device location operations;

FIG. 3C illustrates example reference image rendering surfacesrespectively in a frontal view with glares and in a tilted view;

FIG. 3D illustrates an example configuration in which an augmentedreality graphics superimposition unit uses light map based controloutput to perform graphics superimposition operations;

FIG. 3E illustrates an example configuration in which an ambient lightcontrol unit uses light map based control output to perform ambientlight control operations;

FIG. 4 illustrate example process flows; and

FIG. 5 illustrates an example hardware platform on which a computer or acomputing device as described herein may be implemented.

DESCRIPTION OF EXAMPLE EMBODIMENTS

Example embodiments, which relate to surround ambient light sensing,processing and adjustment for vision devices, are described herein. Inthe following description, for the purposes of explanation, numerousspecific details are set forth in order to provide a thoroughunderstanding of the present invention. It will be apparent, however,that the present invention may be practiced without these specificdetails. In other instances, well-known structures and devices are notdescribed in exhaustive detail, in order to avoid unnecessarilyoccluding, obscuring, or obfuscating the present invention.

Example embodiments are described herein according to the followingoutline:

-   -   1. GENERAL OVERVIEW    -   2. ACQUIRING LIGHT SOURCE INFORMATION    -   3. EXAMPLE LIGHT MAP BASED CONTROLLER    -   4. DISPLAY MANAGEMENT    -   5. DEVICE LOCATION MANAGEMENT    -   6. PERSPECTIVE RENDERING    -   7. AUGMENTED REALITY    -   8. AMBIENT LIGHT CONTROL    -   9. EXAMPLE PROCESS FLOWS    -   10. IMPLEMENTATION MECHANISMS—HARDWARE OVERVIEW    -   11. EQUIVALENTS, EXTENSIONS, ALTERNATIVES AND MISCELLANEOUS

1. General Overview

This overview presents a basic description of some aspects of an exampleembodiment of the present invention. It should be noted that thisoverview is not an extensive or exhaustive summary of aspects of theexample embodiment. Moreover, it should be noted that this overview isnot intended to be understood as identifying any particularlysignificant aspects or elements of the example embodiment, nor asdelineating any scope of the example embodiment in particular, nor theinvention in general. This overview merely presents some concepts thatrelate to the example embodiment in a condensed and simplified format,and should be understood as merely a conceptual prelude to a moredetailed description of example embodiments that follows below. Notethat, although separate embodiments are discussed herein, anycombination of embodiments and/or partial embodiments discussed hereinmay be combined to form further embodiments.

Under techniques as described herein, relatively accurate light sourceand illumination information may be acquired from sensor data generatedby one or more (e.g., a single, a multitude of, etc.) directional imagesensors in one or more of a variety of geographic locations such as ascene where original source images are captured, a mobile device or a(home) TV which forms a temporally closed loop ambient capture anddisplay management setup, a location of a video conference, a colorgrading environment, an image rendering environment, an automobile, atunnel, etc.

The acquired light source and ambient light information may be used toperform display management operations, device location operations,augmented reality superimposition operations, ambient light controloperations, etc., for any of a wide range of devices. These devices mayinclude relatively small footprint devices (e.g., mobile devices, videogame devices, camera phones, etc.), relatively large footprint devices(e.g., home-based systems, desktop computer displays, movie cameras,etc.), display devices (e.g., dashboard displays, rear-seat displays,etc.) in automobiles, display devices operating in conjunction withvideo conferencing systems, display devices used by colorists in colorgrading environments, movie cameras, etc.

Display management operations refer to operations that exploit (e.g.,full, etc.) display capabilities of a target device (e.g., an imagerendering device, etc.) for the purpose of rendering (e.g. visualadjustment, tone and gamut mapping, appearance mapping) images with highdetails, high dynamic range and vivid colors. Under techniques asdescribed herein, these operations may be performed based in part onlight source information acquired in real time or in near real time atan image rendering environment in which images are being rendered.Facial detection or face tracking along with the light sourceinformation may be used to enable the target device to determine spatialrelationships between a viewer and any light sources present in theimage rendering environment. A directional light map may be built for areference image rendering surface of the target device on which imagesare to be rendered. Ambient light properties of the reference imagerendering surface as perceived by the viewer may bededuced/estimated/predicted based at least in part on the directionallight map and optical properties of a display screen to which thereference image rendering surface correspond. Based on the ambient lightproperties, different black levels, different contrast settings,different white points, different primary colors, etc., may beconfigured for different spatial portions of the reference imagerendering surface. For example, for a first spatial portion (of thereference image rendering surface) with a relatively low ambient lightlevel, a normal black level, a normal contrast setting, etc., may beconfigured. Here, the first spatial portion refers to a specific area onthe reference image rendering surface. In contrast, for a second spatialportion (of the reference image rendering surface) with a relativelyhigh ambient light level, an elevated black level, an elevated contrastsetting, etc., may be configured. Here, the second spatial portionrefers to another specific area on the reference image rendering surfaceother than the first spatial portion. As a result, images can berendered with high perceptual fidelity and high immersive-ness for asalient part or the entirety of the image rendering surface in a varietyof image rendering environments.

Device location operations refer to operations that cause a targetdevice (e.g., an image rendering device, etc.) to be repositioned into abetter position and/or a better orientation for the purpose of renderingimages with high details, high dynamic range and vivid colors. Undertechniques as described herein, these operations may be performed basedin part on light source information acquired in real time or in nearreal time at an image rendering environment in which images are beingrendered. With the light source information, directional light maps fora variety of spatial volumes, spatial surfaces, etc., can be deduced inreference to a viewer whose head (or face) is tracked for example byface detection techniques, head tracking techniques, etc. For example,the target device may generate a directional light map for a displayscreen (or a reference image rendering surface) of the target device atthe current position and/or current orientation, as well as one or moredirectional light maps for other reference surfaces (or candidate imagerendering surfaces) to which the display screen of the target device maybe relocated. In response to determining there is an optimal referencesurface for image rendering based on the generated directional lightmaps, if automatically controllable moving mechanisms are available, thetarget device may be automatically repositioned (or relocated) into theoptimal reference surface. Additionally, optionally, or alternatively, aviewer/user of the target device may be alerted (e.g., haptically,visually, audibly, etc.) to the existence of one or more better (e.g.,optimal, etc.) reference surfaces to which the target device should berepositioned. In response, the viewer/user may effectuate therepositioning (or relocation) of the target device to the suggestedreference surface or another better surface other than one at thecurrent position and/or current orientation.

When a display screen is tilted from a view angle (or a line of sight)of a viewer/user for example to avoid strong light reflections on thedisplay screen, an image may be displayed in perspective with theviewing angle of the viewer/user. The rendered portion of the image maybe perceptually the same as or equivalent to a rendered portion of thesame image as if rendered on a non-tilted image rendering surface (as ifthe user would have a frontal view of the non-tilted image renderingsurface). In some embodiments, this type of perceptive rendering of animage on a tilted image rendering surface may be beneficially used whenthe target device has a high resolution screen (e.g., hiDPI or ‘Retina’type, etc.).

Augmented reality operations refer to operations that superimpose anobject or person that does not actually exist in images captured from ascene onto the images. Under techniques as described herein, theseoperations may be performed based in part on light source informationacquired at an image capturing environment (e.g., a scene, etc.) inwhich the images are taken. The light source information may be used toenable the superimposition of the object or person in a realisticmanner. In an example, in the case of a video conferencing system, lightsource information of a location in a video conference may be capturedand used to superimpose an image portion representing a participant notat the location with images captured at the location. Thesuperimposition of the participant with the images may be performedrealistically (e.g. matching color appearance such as white point andlight distribution in the scene) based on the light source informationof the location; as a result, the participant not at the location mayappear in the images as if the participant were one of the participantsactually present at the location. The superimposed images may beprovided to all locations of the video conference as if all theparticipants were meeting in the same location. In another example, oneor more participants of a video game at one or more different locationsmay be superimposed in a video game scene through augmented realityoperations as described herein.

In some operational scenarios, some or all of ambient light conditionssuch as light sources, blinds, transmissive and/or reflective propertiesof display screen, wall, optical polarizers, etc., present in a viewingenvironment, in an image rendering environment, etc., may becontrollable, automatically without user input and/or manually via userinteractions. Light source information acquired under techniques asdescribed herein can be used to control light source properties, lightsource operational states, blinds, transmissive and/or reflectiveproperties of display screen, wall, optical polarizers, etc., to causethe ambient light conditions of the viewing environment, the imagerendering environment, etc., to become optimal for high quality imagerendering.

Additionally, optionally, or alternatively, the ambient light conditionsof the image rendering environment may be controlled to simulate ambientlight conditions of an original color grading environment in which theimages were color graded. For example, feedbacks may be provided by asystem implementing techniques as described herein to aid in theadjustment of ambient light condition in the image rendering environment(e.g., living room, etc.) to create an optimal viewing experienceconsistent with the original color grading environment for effectivelyand optimally preserving the creative intent. Conversely, feedbacks maybe provided by a system implementing techniques as described herein toaid in the adjustment of ambient light condition in a color gradingenvironment to simulate the image rendering environment (e.g., livingroom, etc.) to allow a colorist to obtain an idea how that imagerendering environment will impact on viewing images being graded by thecolorist.

Light source information as described herein can be used not only topredict/estimate ambient light conditions incident on a referencesurface, but also to determine what a viewer/user would see. Forexample, optimal properties (e.g., reflectivity, scattering, glossiness,diffusive properties, etc.) of a display screen may be used togetherwith the ambient light conditions incident on the reference surface todetermine what ambient light conditions would be perceived by theviewer/user. Additionally, optionally, or alternatively, the lightsource information may be used in some image rendering environmentsassociated with a moving target device in a vehicle, with a pedestrian,etc., to determine ambient light or directional light that are notreflected off from a display screen. For example, the light sourcecondition may be used to determine incoming headlights or glares. Amovable/tiltable display screen such as a dashboard display, etc., maybe (e.g., automatically, etc.) moved/tilted to a better position and/ora better orientation to allow a viewer/user to look to a differentviewing angle that would not look into strong light or glares.

In some example embodiments, mechanisms as described herein form a partof a media processing system, including but not limited to any of:mobile device, video game devices, display device, media player, mediaserver, media production system, camera systems, home-based systems,communication devices, video processing system, video codec system,studio system, streaming server, cloud-based content service system, ahandheld device, game machine, television, laptop computer, netbookcomputer, tablet computer, cellular radiotelephone, electronic bookreader, point of sale terminal, desktop computer, computer workstation,computer server, computer kiosk, or various other kinds of terminals andmedia processing units.

Various modifications to the preferred embodiments and the genericprinciples and features described herein will be readily apparent tothose skilled in the art. Thus, the disclosure is not intended to belimited to the embodiments shown, but is to be accorded the widest scopeconsistent with the principles and features described herein.

2. Acquiring Light Source Information

FIG. 1A illustrates an example light map based controller 100 comprisinga light source identification module 102 operating in conjunction withone or more directional image sensors such as one or more cameras 104,one or more non-camera directional image sensors 106, etc.

In some embodiments, some or all of the cameras (104) and the non-cameradirectional image sensors (106) is a part of a computing devicecontaining the light source identification module (102). In someembodiments, some or all of the cameras (104) and the non-cameradirectional image sensors (106) is a part of an accessory to a computingdevice containing the light source identification module (102). In someembodiments, some or all of the cameras (104) and the non-cameradirectional image sensors (106) is external to a computing devicecontaining the light source identification module (102). A directionalimage sensor as described herein may be directly or indirectly linked tothe light source identification module (102) via one or more data links(e.g., wireless, wired, electric contact based, inductively coupled,Bluetooth, USB based, etc.).

A light source identification module as described herein may beimplemented in, but not limited to only, any of: a mobile phone; a videogame device, a television; a tablet computer; a heads up display (HUD)device; a dashboard display device; a display device operating inconjunction with one or more computers; a display device in anautomobile, boat, ship, aircraft, etc.; a home theater system; a colorgrading system; etc.

A directional image sensor (e.g., one of 104, 106, etc.) may beconfigured for wide angle capturing/sensing of directional light. Adirectional image sensor (e.g., one of 104, 106, etc.) may be configuredfor medium or narrow angle capturing/sensing of directional light. Adirectional image sensor (e.g., one of 104, 106, etc.) may be configuredfor generating directional image sensor data of relatively high spatialresolutions. A directional image sensor (e.g., one of 104, 106, etc.)may be configured for generating directional image sensor data ofrelatively low spatial resolutions.

In some embodiments, a directional image sensor as described herein isprovisioned, disposed, secured, etc., on (e.g., edges of, etc.) a caseof a computing device such as a (e.g., protective, etc.) case of amobile phone, an accessory to a video game device, a case of a tabletcomputer, a housing or affixture of a device (e.g., television, set-topbox, etc.), etc.

In some embodiments, the same type of directional image sensors can beused in the light source identification module (102). In someembodiments, two or more different types of directional image sensorscan be used in the light source identification module (102). Forexample, in an embodiment as shown in FIG. 1A, the front and back facingcameras (104) may be directional image sensors of a first type, whereasfour directional image sensors (106) at edges of the light sourceidentification module (102) may be directional image sensors of a seconddifferent type.

In some embodiments, the cameras (104), individually or collectively,comprise software, hardware, a combination of software and hardware,etc., such as an electro-optical configuration including but not limitedto an image acquisition controller, an aperture, one or more lenses,zero or more filters, one or more image sensors, programmatic and/ormanual mechanisms to control settings (e.g., focal length, aperturesize, exposure time, etc.) and operations of one or both of the cameras(104) for directional light sensing purposes. In some embodiments, asillustrated in FIG. 1A, the cameras (104) may represent front and backfacing cameras that are a part of the light source identification module(102), for example in a mobile phone (e.g., an iOS device such asiPhone, an Android device, a Windows phone, etc.). Examples of thecameras (104) may include, but are not limited to only, any of: digitalcameras, non-digital cameras, analog camera, camera equipped withphotosensitive chemicals, webcams, etc.

Additionally, optionally, or alternatively, a directional image sensoras described herein may be, but is not limited to only, any of:CMOS-based directional image sensors, CCD-based directional imagesensors, nanotube-based directional image sensors (e.g., a combinationof nanotubes that are oriented to different spatial directions, etc.),LED-based image sensors, solar cell arrays, quantum dot photoconductors,photodiodes, polymer-based image sensors (e.g., used on a case, etc.),any combination of the foregoing, etc. Some or all of the directionalimage sensors may be calibrated to generate relatively reliableluminance and colorimetric information.

A light map based controller (e.g., 100 of FIG. 1A) as described hereinmay be deployed to operate in diverse environments such as homes,offices, vehicles, highways, tunnels, roads, public places, moviestudios, color grading environments, color grading studios, ambientlight controlled environments, ambient light uncontrolled environment,etc.

FIG. 1B illustrates an example operational environment 108 in which thelight map based controller (100) may operate. The operationalenvironment (108) comprises a physical structure 110 (e.g., a room, anoffice, a building, etc.) in which a user 118 may make use of a computerdevice such as a mobile phone, a video game device, etc., that operatesin conjunction with the light map based controller (100).

Zero, one or more light sources may be present in any given operationalenvironment such as 108 of FIG. 1B. A light source (e.g., emissive,reflective, transmissive, etc.) as described herein may also be emittingor reflecting off light in different light intensities, differentcolors, etc. Additionally, optionally, or alternatively, a light sourceas described herein may be any geometric shapes including but notlimited to only any of: point light sources, area light sources (ornon-point light sources), textured light sources, array-based lightsources, edge-based light sources, etc.

Light sources may be located in various directions, various distances,etc., from a reference image rendering surface (e.g., 140 of FIG. 1A,etc.). The reference image rendering surface may be a display screen ofa target device such as a mobile phone; a video game device; atelevision; a tablet computer; a heads up display (HUD) device; anAugmented Reality (AR) Display; a dashboard display device; a displaydevice operating in conjunction with one or more computers; a displaydevice in an automobile, boat, ship, aircraft, etc.; a home theatersystem; a color grading system; etc. In a non-limiting example, thereference image rendering surface may be a part of a computing devicecontaining the light source identification module (102).

In the operational environment (108) as illustrated in FIG. 1B, an arealight source 112—which may be provided by an object or a wall of highlight reflectivity or albedo—is present to the left of the light mapbased controller (100); a point light source 114—which may be providedby an active light emitter (e.g., a white or color LED, etc.)—is presentat an upper left direction relative to the light map based controller(100); a portion of sunlight from the Sun (116) may also reach into thephysical structure (110); etc.

In some embodiments, directional image sensors of the light map basedcontroller (100) collectively represent an implementation of a camerasubsystem able to capture light source images each of which may begenerated based on a combination of multiple portions of directionalimage sensor data acquired by these directional image sensors. Examplesof light source images include, but are not limited to only: any of: lowspatial resolution light source images, medium spatial resolution lightsource images, high spatial resolution light source images, sphericalimages, non-spherical images, 3D images, etc.

Each portion of the multiple portions of directional image sensor thatare used to generate a light source image may represent a portion ofdirectional image sensor data captured by a directional image sensor ofthe directional image sensors for one or more solid angles relative tothe directional image sensor or an optical axis of the directional imagesensor. In a first non-limiting example, a portion of directional imagesensor data may be generated by a directional image sensor from electriccharges in diodes as excited/produced by light incident on thedirectional image sensor from one or more solid angles in one or moredirections relative to the directional image sensor or an optical axisof the directional image sensor. In a second non-limiting example, aportion of directional image sensor data may be generated by adirectional image sensor and comprises pixel data representing lightincident on the directional image sensor from one or more solid anglesof one or more directions relative to the directional image sensor or anoptical axis of the directional image sensor.

FIG. 1C illustrates an example light source image 130, which may begenerated by the light map based controller (100) of FIG. 1A or FIG. 1Bin the example operational environment (108) of FIG. 1B. In an example,a light source map image as described herein may be generated based ondirectional image sensor data acquired through one or two fisheye lenscameras (e.g., the front and back facing cameras (104), etc.) in thefront and back of a mobile device containing the light sourceidentification module (102). In another example, a light map image asdescribed herein may be generated based on directional image sensor dataacquired through directional image sensors that are in addition to or inplace of one or two cameras. Some of the directional image sensors maybe non-camera directional image sensors deployed at different edges of aphysical housing or a physical case containing the light sourceidentification module (102); thus, because of these non-cameradirectional image sensors, the viewing angles of any existing camerasneed not be as wide as that of a fisheye lens. Additionally, optionally,or alternatively, in some embodiments, some or all of a light sourceimage as described herein may be entirely generated based on directionalimage sensor data acquired through directional image sensors other thancameras; these non-camera directional image sensors can be deployed onone or more edges of the light source identification module (102) suchas a mobile device, etc. In some embodiments, some of the directionalimage sensors may be provisioned redundantly to allow light blockage tosome image sensors without adversely impacting the construction of alight source image as described herein in various operational scenarios.

In some embodiments, the light source image (130) may, but needs not to,be of a high spatial resolution. In some embodiments, as the lightsource image (130) is to capture directions and positions of lightsources that can affect image rendering qualities on an image renderingsurface such as the reference image rendering surface (140 of FIG. 1A),a few hundred to thousand pixels may be sufficient to represent theselight sources and their directions. To reduce electricity consumption(or to save battery resources), image sensors may be binned or pixels,photo cells, etc. selectively read out so that the image sensors may begenerating a data output from every 10^(th) pixel, every 100^(th) pixel,etc., when operating in a special mode for acquiring directional imagesensor data, rather than generate a data output from every pixel whenoperating in a normal picture capturing mode. Additionally, optionally,or alternatively, spatial and/or or temporal interpolation may beperformed based on acquired data at some pixels to fill in, predict orestimate (e.g., pixel-level) light source related information gaps atsome other pixels, as pixel level information may not be available forthe other pixels (for example) because of temporary or permanentblockage of one or more image sensors. As a result, continuous capturingof megapixel images that may drain battery very quickly may be avoidedunder techniques as described herein. Additionally, optionally, oralternatively, special purpose integrated circuits, coprocessors,analog-to-digital converters, lenses, shutters, etc., that exhibitrelatively high energy efficiency may be used to implement at least someof the techniques as described herein.

As shown in FIG. 1C, the light source image (130) may be a sphericalimage comprising a first light source image portion 122, a second lightsource image portion 124, a third light source image portion 126, etc.,which respectively represent the area light source (112), the pointlight source (114), the Sun (136), etc. By way of example but notlimitation, The spherical light source image (130) may be represented ina three dimensional spatial coordinate system; by way of example but notlimitation, the spatial coordinate system may be formed based on spatialcoordinates x, y, and z, as illustrated in FIG. 1C, of an overall fourdimensional space time coordinate system formed based on space and timecoordinates x, y, z and t (representing the time coordinate; not shown).Additionally, optionally, alternatively, instead of using an X-Y-Zcoordinate system, another coordinate system including but not limitedto a spherical coordinate system based on longitude and latitude, acoordinate system based on azimuthal and elevation angles, etc., can beused for representing a light source image therein.

In some embodiments, the spatial coordinate system in which the lightsource image (130) is represented may be directly defined in relation toa spatial position, a spatial orientation, etc., of a reference surfacesuch as the reference image rendering surface (140 of FIG. 1A) at a timewhen the time directional image sensor data for generating the lightsource image (130) is acquired by the directional image sensors of thelight map based controller (100). In these embodiments, the spatialcoordinate system may be called as a relative coordinate system, as thespatial coordinate system is defined relative to the spatial position,the spatial orientation, etc., of the reference surface.

In some embodiments, instead of or in addition to containing pixel data,a light source image may be represented by, or may be converted to,(e.g., geometric, optical, physical, etc.) constructs that are derivedfrom the pixel data combined/assembled from directional image sensors.In some embodiments, some or all of a light source image as describedherein may be represented by vectors, contours, point spread functions,primary colors, etc., related to light sources detected at least in partfrom sensor data from directional image sensors. For example, instead ofcontaining pixel data, the light source image (130) may comprise a firstvector 132 that represents the first light source (112), a second vector134 that represents the second light source (114), a third vector 136that represents the third light source (116), etc. In some embodiments,each of these vectors (e.g., 132, 134, 136, etc.) may be a specific unitvector on a unit sphere in relation to the reference point 120. Otherconstructs such as contours, point spread functions, primary colors,etc., may also be used in the light source image (130) to representshapes, light distribution properties, colorimetric properties such aswhite points, etc., of the light sources (e.g., 112, 114, 116, etc.) inthe light source image (130). In some embodiments, techniques such asthose related to medium cut algorithm may be used to convert a lightsource image in a pixel data representation (e.g., pixels in a sphericalimage, etc.) to a new light source image in a non-pixel datarepresentation that uses non-pixel data constructs. These techniques maybe used to identify highlights and/or bright parts of a spherical imageand generate a vector representation comprising a set of vectorcoordinates identifying positions of the highlights and/or bright parts.The set of vector coordinates may comprise combinations/coordinates oflongitude and latitude. Some discussion of medium cut algorithms can befound in P. Debevec, “A Median Cut Algorithm for Light Probe Sampling,”Technical Report 67, USC Institute for Creative Technologies (2005), thecontent of which is incorporated by reference herein.

In some embodiments, the spatial coordinate system in which the lightsource image (130) is represented may be directly or indirectly definedin relation to a relatively universal reference coordinate system suchas a world coordinate defined in relation to the Earth's position andorientation, the solar system's spatial location, spatial orientation,etc., at a time when the time directional image sensor data forgenerating the light source image (130) is acquired by the directionalimage sensors of the light map based controller (100). In theseembodiments, the spatial coordinate system may be called as an absolutecoordinate system, as the spatial coordinate system is defined relativeto the spatial position, the spatial orientation, etc., of therelatively universal reference coordinate system.

In some embodiments, while the spatial coordinate system in which thelight source image (130) is represented may be directly defined inrelation to a spatial position, a spatial orientation, etc., of areference surface, a spatial relationship (e.g., represented by atransformation matrix, a set of algebraic relationships, etc.) may beestablished between the spatial coordinate system and a relativelyuniversal coordinate system. The spatial relationship between the twocoordinate systems may be used to map between a spatial location, aspatial displacement, a vector, etc., in one of the two coordinatesystems to those in the other of the two coordinate systems. Forexample, a spatial position, a spatial orientation, etc., of the sun (oranother object such as another light source, a face, a referencesurface, a light reflection area on a reference surface, etc.) in a wordcoordinate system may be converted into a corresponding spatialposition, a spatial orientation, etc., of the sun (or the other objectsuch as the other light source, the face, the reference surface, thelight reflection area, etc.) in the relative coordinate system definedrelative to the reference surface. Conversely, a spatial position, aspatial orientation, etc., of the reference surface (or another objectsuch as a light source, a face, a light reflection area on a referencesurface, etc.) in the relative coordinate system may be converted (e.g.,inversely, etc.) into a corresponding spatial position, a spatialorientation, etc., of the reference surface (or the other object such asthe light source, the face, the light reflection area, etc.) in theuniversal coordinate system.

In some embodiments, the light map based controller (100) may operate inconjunction with one or more locational sensors such as one or more of(e.g., internal, etc.) motion sensors, position sensors, orientationsensors, gyroscopes, electronic compasses, accelerometers, globalpositioning system (GPS) modules, etc. In some embodiments, some or allof these locational sensors may be built-in components of the light mapbased controller (100). Some or all of geographic information, motioninformation, position information, velocity information, etc., acquiredby these locational sensors may be used to set up a reference origin 120(e.g., in a central portion of the reference surface, etc.), a spatialorientation, etc. of the spatial coordinate system in which the lightsource image (130) is represented.

In some embodiments, it is not necessary to represent the light sourceimage (130) in relatively universal reference coordinate system. In someembodiments, for a stationary display device such as a stationarytelevision, a stationary computer display, etc., it is adequate torepresent the light source image (130) in a coordinate system inrelation to spatial properties of the stationary display device.

The light source image (130) may comprise light source information suchas related to some or all of spatial positions, spatial directions,intensities, colorimetric properties, etc., related to each of some orall of the light sources represented in the light source image (130).

In an example, the light map based controller (100) is configured tocombine/assemble directional image sensor data (e.g., pixel data, etc.)collected by the directional image sensors to generate a two dimensionalspherical image, a three dimensional image, etc.; apply a median cutfilter to the generated image to identify light sources and to determinelight source information such as related to some or all of locationalinformation (e.g., spatial position, spatial orientation, longitude,latitude, distance derived from 3D diversity information, etc.),intensities, colorimetric properties, etc., related to each of some orall of the identified light sources; generate a light source image basedon the light source information; etc. Some or all of these foregoingoperations may be performed repeatedly, every 100 milliseconds, every500 milliseconds, every second, on demand, in response to external orinternal events/clock signals, etc.

In some embodiments, the light source image (130) may be built withpixel information that represents relatively bright pixels or brightimage portions in the directional image sensor data. Thus, thedirectional image sensor data may, but is not required to, be of a veryhigh dynamic range. Luminance information in the directional imagesensor data can be relative (e.g., a codeword which is not luminancevalue but can be mapped to a luminance value) or absolute (e.g. aluminance value in cd/m² or the like). Additionally, optionally, oralternatively, the directional image sensor data may, but is notrequired to, be captured with high frame rates or low ISO noise, withunderexposure using a low dynamic range (LDR) camera, etc.

The directional image sensor data can be captured with differentexposure values (EV) and interleaved for gathering light sourceinformation in relatively high portions, etc., of a visual dynamic range(e.g., 1000 nits, 5000 nits, 12,000 nits, 20,000 nits, a luminance valueof an extremely bright object, etc.). For example, some of thedirectional image sensor data may be captured every even times with afirst exposure value suitable for capturing highlights, whereas someother of the directional image sensor data may be captured every oddtimes with a second exposure value suitable for capturing dark imageportions. The directional image sensor data captured with differentexposure values may be used together to identify where light sourcesare, where there exist no light sources, what light source informationpertains to the light sources, etc. In some embodiments, it isacceptable to underexpose (clip the darks) the directional image data toguarantee the capture of all the highlight information (e.g. highlightcusps and textures) if the sensors involved with image capture have alimited dynamic range.

Additionally, optionally, or alternatively, techniques based on assortedpixels may be applied to trade resolution for dynamic range for thepurpose of identifying light sources and determining light sourceinformation of the identified light sources. Some examples of assortedpixel based techniques are described in S. K. Nayar and T. Mitsunaga,“High Dynamic Range Imaging: Spatially Varying Pixel Exposures,” IEEEConference on Computer Vision and Pattern Recognition (CVPR), Vol. 1,pp. 472-479 (June, 2000), the contents of which are incorporated byreference herein.

In some embodiments, the light source image (130) comprises locationalinformation 128 of a viewer (e.g., 118 of FIG. 1B, etc.) such aslocational information of the viewer's head, face or eyes. The viewer(118) may represent a user who is viewing images rendered on a referenceimage rendering surface on a mobile phone, a video game device, a tabletcomputer, a television, etc., that implements the light map basedcontroller (100). The locational information (128) of the viewer mayinclude one or more of distance, angles, etc., of the viewer's face oreyes in relation to a reference point such as the reference origin(120), a central point of the reference image rendering surface, anormal direction of the reference image rendering surface at a centralpoint of the reference image rendering surface, etc.

In some embodiments, the locational information (128) of the viewer maybe computed or analyzed by the directed light map acquisition device(100) from a rough silhouette captured in one or more portions of thedirectional image sensor data. The locational information (128) can thenbe used by the light map based controller (100) to predict locations oflight reflection areas caused by light source reflections on a (e.g.,reflective, etc.) screen such as the reference image rendering surface,etc., on which images are rendered and viewed by the viewer (118).

In some embodiments, the directed light map acquisition device (100) isconfigured to acquire and maintain other data such as other types ofsensor data, non-sensor data, reference data such as a sunlightdirection at a given location at a given time, weather (overcast orsunny) at a given location at a given time, etc. The other data acquiredand maintained by the directed light map acquisition device (100) can beacquired by the directed light map acquisition device (100) beforehand(e.g., before using the information to improve image rendering quality,etc.) or contemporaneously (e.g., while using the information to improveimage rendering quality, etc.). Additionally, optionally, oralternatively, the directed light map acquisition device (100)—which forexample may be implemented by a stationary device such as a television,etc.—can be configured to analyze the illumination trend from availableand/or acquired sensor data, non-sensor data, etc., over a historicaltime period such as past several days, past several hours, past severalminutes, etc., to estimate its location in a room, to estimate lightsource information, etc.

3. Example Light Map Based Controller

FIG. 2 depicts a block diagram of an example light map based controller(e.g., 100 of FIG. 1A or FIG. 1B, etc.) comprising a light sourceidentification module (e.g., 102 of FIG. 1A, etc.), a light map-basedcontrol unit 210, etc. In some embodiments, the light sourceidentification module (102) comprises a sensor data input unit 202, aface tracking unit 204, a data repository 206, a light source image unit208, etc. Each of these modules, units, etc., in the light map basedcontroller (100) may be implemented in software, hardware, a combinationof software and hardware, etc.

The light map based controller (100), or the sensory data input unit(202) therein, can be configured to receive sensor data 212 (e.g., realtime sensor data, near real time sensor data, etc.) from one or more ofdirectional image sensors (e.g., 104 or 106 of FIG. 1A, etc.),locational sensors (e.g., motion sensors, position sensors, orientationsensors, gyroscopes, electronic compasses, accelerometers, GPS modules,etc.), etc.

In some embodiments, the data repository (206) represents one or moredatabases, one or more data storage units/modules/devices, etc.,configured to support operations such as storing, updating, retrieving,deleting, etc., with respect to sensor data (e.g., 212, etc.),non-sensor data, reference data, etc.

The light map based controller (100), or the face tracking unit (204)therein, can be configured to receive one or more portions ofdirectional image sensor data. The one or more portions of directionalimage sensor may be a part of the sensor data (212) received andforwarded by the sensory data input unit (202). The face tracking unit(204) may be configured to apply face tracking techniques (e.g., basedon silhouette, highlights, light reflections, etc., that arecharacteristics of a human face, etc.) to analyses of the one or moreportions of directional image sensor data. Based on results of theanalyses, the face tracking unit (204) may be configured to track (e.g.,in real time, in near real time, etc.) the location of a face of aviewer (e.g., 118 of FIG. 1C, etc.) in relation to a reference surfacesuch as a reference image rendering surface, etc.

The light map based controller (100), or the light source image unit(208) therein, can be configured to receive directional image sensordata (some portions of which may also be used for face trackingpurposes). The directional image sensor may be a part of the sensor data(212) received and forwarded by the sensory data input unit (202). Thelight source image unit (208) may be configured to analyze thedirectional image sensor data, identify overlaps in portions of thedirectional image sensor data from different directional image sensors,generate a light source image (e.g., 130 of FIG. 1C, in real time, innear real time, etc.) by applying light source detection techniques(e.g., medium cut, assorted pixels, etc.) the directional image sensordata, etc.

The light map based controller (100), or the light map-based controlunit (210) therein, can be configured to receive the locationalinformation of the viewer (118) as generated by the face tracking unit(204), the light source image (130) as generated by the light sourceimage unit (208), retrieve other data such as reference data (e.g., asunlight direction, weather, in real time, in near real time, etc.),etc. In some embodiments, the light source image (130) may comprise thelocational information of the viewer (118), the locational informationof the reference surface (e.g., the reference image rendering surface140 of FIG. 1A, etc.), etc.

Based on some or all of the locational information of the viewer (118),the location information of the reference surface, the light sourceimage (130), the reference data, etc., the light map based control unit(210) may be configured to construct a directional light map of thereference surface. The directional light map of the reference surfacemay represent a temporal spatial prediction/estimation of how ambientlight is incident on and distributed over a plurality of differentspatial portions of the reference surface. The directional light map maypredict (e.g., based on ray tracing with respect to light from lightsources, etc.) one or more of a specific ambient light direction, aspecific ambient light intensity, specific ambient light colorimetricproperties, etc., for any given point on some or all portions of thereference surface. The ambient light properties at various spatialportions of the reference surface may be estimated based on light sourceinformation in the light source image, reference data (e.g., the sun'sposition, weather conditions or forecasts applicable to the referencesurface, etc.), etc. In some embodiments, a directional light map asdescribed herein may comprise some or all of ambient light informationon a reference surface that is computed/predicted/estimated based atleast in part on a light source image without actual physicalmeasurements of ambient light at the reference surface. Additionally,optionally, or alternatively, more than one directional light map may becomputed/predicted/estimated based at least in part on a light sourceimage at any given time. For example, a set of one or more directionallight maps may be computed/predicted/estimated for a set of one or morereference surfaces for any given time based at least in part on a lightsource image.

For example, a first directional light map in the set of directionallight maps may be computed/predicted/estimated for a first referencesurface in the set of reference surfaces based at least in part on alight source image for the given time. A second directional light map inthe set of directional light maps may be computed/predicted/estimatedfor a second reference surface in the set of reference surfaces based atleast in part on the same light source image for the same given time.The first reference surface may be a reference image rendering surfacethat is an actual image rendering surface of a display device at thegiven time, whereas the second reference surface may be a referenceimage rendering surface that is not the actual image rendering surfaceof the display device at the given time. For example, the secondreference surface may be an optimal image rendering surface to be at forthe display device in order to achieve optimal image rendering results(e.g., relatively detailed images, relatively vivid colors, etc.).

Additionally, optionally, or alternatively, the light map basedcontroller (100), or the light map-based control unit (210) therein, canbe configured to generate light map based control output 214 based onthe directional light map of the reference surface, and/or some or allof the data used to construct, predict, or estimate the directionallight map of the reference surface.

In some embodiments, some or all of sensor data, light source images,positional information of a view's face, directed light maps withrespect to reference surfaces, reference data, etc., are generated oraccessed in real time, in near real time, etc. For example, some or allof sensor data as described herein may be acquired in real time throughsensors of a target device (e.g., an image rendering device, a moviecamera system, a colorist workstation, an automobile-based displaydevice, etc.). A directed light map is computed in near real time (e.g.,within a small time window of 10 milliseconds, 20 milliseconds, 100milliseconds, etc., from an earliest time at which the sensor data iscollected) for a spatial volume, a spatial surface (e.g., an imagerendering surface of an image rendering device, etc.), etc. The directedlight map is derived as a function of an image source image generatedbased at least in part on the sensor data in real time or in near realtime. Directional light maps can be continuously (e.g., every 100milliseconds, every 5 seconds, etc.) constructed.

In a first example, a viewer walking in an alley of changing light anddark levels, a target device (e.g., an image rendering device, a mobilephone, a video game device, a tablet computer, etc.) with the viewer maycontinuously generate spatial temporal directional light maps at aplurality of time points in real time. Even when the viewer's hand thatis holding the device may cover up some sensors with the device (or acover thereof), the device may be configured to continuously performface tracking (or facial detection), assemble available sensorinformation to light source images, and further construct directionallight maps from the light source images, time-varying locationalinformation of the viewer, etc.

In a second example, a target device (e.g., an image rendering device,etc.) may be configured to perform history tracking and maintain/storehistorical data (including but not limited to trend data) of some or allof light source information, face tracking information, devicepositions, device orientations, etc., for one or more time intervals(e.g., 30 seconds, five minutes, etc.). The device may be configured togenerate estimates/predictions of some or all of light sourceinformation, face tracking information, device positions, deviceorientations, etc., at any given time (e.g., in real time, in near realtime, etc.) based on the historical data. For instance, at a time whensome sensors of the device are blocked and some information derived fromwhat would have been collected by the sensors is hidden or missing, thehistorical information collected by the device for a preceding timeinterval to the time may be used to predict or estimate the hidden ormissing information, until the sensors are unblocked and information canagain be derived from what is collected by the sensors.

Real time or near real time information regarding light sources, ambientlight conditions, etc., under techniques as described herein can berepeatedly (e.g., continuously, periodically, etc.) derived, and used indiverse operational scenarios of a target to perform a variety ofimage-related operations (e.g., improve image rendering qualities, etc.)in real time or in near real time.

4. Display Management

FIG. 3A illustrates an example configuration in which a display manager302 uses light map based control output such as 214 of FIG. 2 to performdisplay management operations in an image rendering environment.

In some embodiments, the display manager (302) comprises software,hardware, a combination of software and hardware, etc., configured toreceive image content from an image content source 304, receive thelight map based control output (214), perform display managementoperations in rendering the image content on a reference image renderingsurface (e.g., 140 of FIG. 1A, etc.) based at least in part on the lightmap based control output (214), etc.

In some embodiments, the light map based control output (214) comprisesa directional light map for the reference image rendering surface at agiven time when a specific image is to be rendered on the referenceimage rendering surface (140) by the display manager (302). Based onlight source information acquired from the image rendering environment,the display manager (302) may be configured todetermine/predict/estimate (e.g., various, a plurality of, etc.) ambientlight properties such as light levels, light colorimetric properties,etc., at some or all locations on the reference image rendering surface(140) in the image rendering environment for the given time based on adirectional light map in the light map based control output (214). Insome embodiments, the display manager (302) is configured to determineoptical properties such as light reflective properties, light scatteringproperties, light diffusion properties, glossiness, etc., of a displayscreen (e.g., corresponding to the reference image rendering surface,etc.), build bidirectional scattering distribution functions (BSDFs)based on the optical properties, predict/estimate the ambient lightproperties as perceived by a viewer based at least in part on thedirectional light map, the optical properties, the BSDFs, etc.

As a part of the display management operations performed by the displaymanager (302) in rendering the specific image on the reference imagerendering surface (140), the display manager (302) may be furtherconfigured to set different black levels, different contrast settings,different white points, different primary colors, etc., in differentportion of the specific image on the reference image rendering surface(140) based on different ambient light properties predicted from thedirectional light map. In some embodiments, some or all of black levels,contrast settings, white points, primary colors, etc., in variousportions of the specific image may be computed as functions of theambient light conditions of various portions of an image renderingsurface, where the various portions of the image rendering surfacerespectively correspond to the various portions of the specific image.As used herein, a portion of an image may refer to one or more of:pixels, pixel blocks, regular shapes each comprising a plurality ofpixels, irregular shapes each comprising a plurality of pixels, etc.

For example, the directional light map may comprisepredictions/estimations for a relatively low ambient light level (e.g.,corresponding to a light reflection of a relatively low intensity, etc.)at a first surface portion 308-1, a medium ambient light level (e.g.,corresponding to a light reflection of a medium intensity, etc.) at asecond surface portion 308-2, a relatively high ambient light level(e.g., corresponding to a light reflection of a relatively highintensity, etc.) at a third surface portion 308-3. Based on thesepredictions/estimations of the directional light image, the displaymanager (302) may be configured to set a relatively low black level, anormal contrast setting, etc., in the first surface portion (308-1), seta medium black level, a higher than the normal contrast setting in thesecond surface portion (308-2), set a relatively high black level, amuch higher than the normal contrast setting in the third surfaceportion (308-3), etc.

Additionally, optionally or alternatively, the directional light map maycomprise predictions/estimations for ambient light colorimetricproperties at the first surface portion (308-1), the second surfaceportion (308-2), the third surface portion (308-3), etc. Based on thesepredictions/estimations of the directional light image, the displaymanager (302) may be configured to set different color appearances(e.g., through different white points, different primary colors, etc.)in the first surface portion (308-1), the second surface portion(308-2), the third surface portion (308-3), etc.

In some embodiments, at least some of the display management operationsperformed by the display manager (302) in dependence on the directedlight map may be performed (e.g., without altering pixel values of thespecific image, in combination with altering pixel values of thespecific image, etc.) through global dimming, through local dimming, byadjusting backlight at or near different portions of the image renderingsurface, etc. In some embodiments, at least some of the displaymanagement operations performed by the display manager (302) independence on the directed light map may be performed through tonemapping, inverse mapping, etc., that alters pixel values of the specificimage at different portions of the specific image that respectivelycorrespond to different portions of the image rendering surface.

Additionally, optionally, or alternatively, the light map based controloutput (214) comprises ambient light information other than thosecomputed for the reference image rendering surface (140). In an example,the light map based control output (214) may comprise light sourceinformation that indicates a strong light (e.g., an incoming vehicle'sheadlight, sunlight, etc.) is directing towards the viewer's sightwithout being reflected from the reference image rendering surface(140). In some embodiments, the reference image surface (140) may beautomatically repositioned or tilted to a location at which the stronglight is not directing towards the viewer's sight. In another example,the light map based control output (214) may comprise light sourceinformation that can be used to determine or generate perceptual effectof the background and surround behind the reference image renderingsurface (140). For example, a system as described herein may determinethat the reference image rendering surface (140) (e.g., a display screenor the like) has a bright green background that generates a perceptualeffect of making the display screen to look different than a darker red.This can be computed by the system using, for example, a color or imageappearance model.

5. Device Location Management

FIG. 3B illustrates an example configuration in which a display manager(e.g., 302 of FIG. 1A, etc.), a device location manager 312, etc., uselight map based control output such as 214 of FIG. 2 to perform one ormore of display management operations, device location operations, etc.

In some embodiments, the device location manager (312) comprisessoftware, hardware, a combination of software and hardware, etc.,configured to receive image content from an image content source 304,receive the light map based control output (214), perform locationmanagement operations that relocates, reorient, etc., a reference imagerendering surface (e.g., 140 of FIG. 1A, etc.) based at least in part onthe light map based control output (214), etc.

In some embodiments, the light map based control output (214) comprisestwo or more directional light maps for two or more reference surfaces ata given time when a specific image is to be rendered on the referenceimage rendering surface (140) by the display manager (302). Thedirectional light maps for the reference surfaces may include a firstdirectional light map for the reference image rendering surface (140), asecond directional light map for a second reference surface other thanthe reference image rendering surface (140). The first directional lightmap may comprise predictions/estimations of intense directional light(e.g. sunlight) hitting some portions of a display screen represented bythe reference image rendering surface (140). The second referencesurface may be an optimal image rendering surface to be at for thereference image rendering surface (140) in order to achieve optimalimage rendering results (e.g., relatively detailed images, relativelyvivid colors, etc.).

In some embodiments, the light map based control output (214), or thesecond directional light map therein, comprises locational informationof the second reference surface. Based at least in part on thelocational information (e.g., a position, a surface orientation, etc.)of the second reference surface, the location manager (312) may beconfigured to cause the reference image rendering surface (140) toreposition itself into (e.g., at, near, within specific error thresholdsfor positions and orientations from, etc.) the second reference surface.

In a first non-limiting example, based at least in part on thelocational information (e.g., a position, a surface orientation, etc.)of the second reference surface, the location manager (312) may beconfigured to generate locational control output to one or moremechanical components (e.g., actuators, motors, movable parts, rotatableparts, etc.) to automatically relocate to the reference image renderingsurface (140) to the second reference surface which may be constrainedto a range of positions, orientations, etc., that are supported by theone or more mechanical components. This may be done for example for atelevision in a home theater environment, for a dashboard display in anautomobile (e.g., to avoid glares on the dashboard display caused byincident light from a light source such as a headlight of a car frombehind, etc.), for a display device in a backseat area of a minivan,etc.

In a second non-limiting example, based at least in part on thelocational information (e.g., a position, a surface orientation, etc.)of the second reference surface, the location manager (312) may beconfigured to generate user interface output to inform a user of theexistence of the second reference area to which the reference imagerendering surface (140) may be optimally relocated. Examples of userinterface output may include, but are not limited to only, any of: oneor more audible alerts (e.g., buzzes, etc.), haptic alerts (e.g.,vibrations, etc.), visual alerts (e.g., visual indicators rendered(e.g., near edges, graphic overlays, transparent overlays, etc.) on thereference image rendering surface (140), etc. In some embodiments,vibration actuators/motors distributed with a target device (e.g., amobile phone, a video game device, etc.) may guide a viewer/userhaptically (e.g., via feeling the vibrations, etc.) to reposition thetarget device to the best viewing angle, the best viewing position, etc.

In some embodiments, the user interface output may provide feedbacks tothe viewer/user for better positioning, for better reorientation, etc.,of the reference image rendering surface. The feedbacks may include, butare not limited to only, variable user perceptible information (e.g.,volume, audible frequencies, colors, light flashing, etc.) about aseverity level in image rendering quality deterioration associated withthe current position of the reference image rendering surface, about anextent of improvement in image rendering quality improvement associatedwith the second reference surface, etc.

In some embodiment, one or more arrows such as 310-1, 310-2, which maybe straight arrows, clockwise circular arrows, counterclockwise circulararrows, etc., can be (e.g., continuously, etc.) rendered on thereference image rendering surface (140) until the user turns off devicepositioning alerts (e.g., similar to turning off an actively buzzingalarm on a mobile phone, etc.), or until the reference image renderingsurface (140) is repositioned into (e.g., at, near, within specificerror thresholds for positions and orientations from, the secondreference surface, a location at which glares disappear, etc.) a finallocation where ambient light conditions on the reference image renderingsurface are optimal or acceptable. Perceptual cues (e.g., alarm sounds,an audible or visual message such as “move to the left,” “tilting to theright”, etc.) for repositioning, reorientation, etc., may be given tothe viewer as the viewer is adjusting the reference image renderingsurface, until the reference image rendering surface (140) isrepositioned into the final location.

Additionally, optionally, or alternatively, the display manager (302)may be configured to determine (e.g., various, a plurality of, etc.)ambient light properties such as light levels, light colorimetricproperties, etc., at some or all locations on the reference imagerendering surface (140) based on a directional light map (e.g.,continuously provided, etc.) in the light map based control output (214)of the reference image rendering surface (140), and performscorresponding display management operations based on these ambient lightproperties, at one or more time points while the reference imagerendering surface (140) is being repositioned, reoriented, etc.

6. Perspective Rendering

In some embodiments, at a location to which the reference imagerendering surface (140) is adjusted, the viewer has a frontal view ofthe reference image rendering surface (140). For example, the viewer'sface may be located at or around a normal direction (e.g., at a centralportion, etc.) of the reference image rendering surface (140); the tiltangle between a line of sight of the viewer and the normal direction iszero degree, or alternatively within a normal range of 5 degrees, 10degrees, etc. However, in some scenarios, at a location to which thereference image rendering surface (140) is adjusted, the viewer may nothave a frontal view of the reference image rendering surface (140); thetilt angle between the line of sight of the viewer and the normaldirection is not zero degree, or alternatively exceeds a normal range of5 degrees, 10 degrees, etc. As used herein, a line of sight (of theviewer) may represent an imaginary line from the viewer's face to (e.g.,a central point of, etc.) the reference image rendering surface (140).

FIG. 3C (a) illustrates an example reference image rendering surface(140) initially at a location at which the user has a frontal view butwith a relatively high intensity glare. FIG. 3C (b) illustrates the samereference image rendering surface (140) that has been adjusted to alocation at which there is no high intensity glare but the user nolonger has a frontal view.

In some embodiments, the display manager (302) may be configured todetermine a tilt angle for the line of sight (of the viewer) from theviewer's face to (e.g., a central point of, etc.) the reference imagerendering surface (140), and render at least a portion (e.g., a majorityportion, a salient portion, 90%, 80%, etc.) of the image, where therendered portion of the image is in perspective with the line of sightof the viewer. As used herein, a rendered portion of an image, asrendered on a tilted image rendering surface, in perspective with a lineof sight may mean that the rendered portion of the image is perceptuallythe same or equivalent to a rendered portion of the same image asrendered on a non-tilted image rendering surface (the user would have afrontal view of the non-tilted image rendering surface. This type ofperceptive rendering of an image on a tilted image rendering surface maybe beneficial when the target device has a high resolution screen suchas a hiDPI or “Retina” type screen. A pixel on an image to be renderedon a non-tilted image rendering surface may be stretched (e.g.,upsampled, interpolated, etc.) to more than one pixel (e.g., 1.2 pixels,1.5 pixels, 2 pixels, etc.) for the purpose of compensating the tiltangle of a tilted image rendering surface. Additionally, optionally, oralternatively, as a part of perceptual rendering, geometric adjustmentsmay be made to rotate the rendered portion of the image on the tiltedreference image rendering surface (140). In some embodiments, whenperspective rendering operations are performed and the rendered portionof the image may no longer be aligned with the physical boundary of thetilted image rendering surface (140), some portions (e.g., one or moretriangular portions, one or more bars, etc.) of the display screen maybe painted black or some other specific colors/patterns/textures, asillustrated in FIG. 3 (b).

Perspective rendering operations as described herein can be performed bythe target device in real time or in near real time. For example, whilea viewer sits in a car, a dashboard display (or an instrument panel) maybe tilted by a target device that controls image rendering on thedashboard display. The target device may detect, predict, estimate,etc., from a directional light map, light source information in a lightsource image, a GPS assisted estimation of ambient light, the actualBSDF properties of the display, etc., that strong glares occur or are tooccur on the dashboard display. Some or all of the directional lightmap, the light source information, the light source image, the GPSassisted estimation of ambient light, etc., may be generated or derivedat least in part based on information acquired from one or more ofsurround cameras (e.g., for parking related operations or collisionwarnings, etc.), GPS units/modules/devices, other motion sensors,reference road maps, etc. In response to detecting that strong glaresoccur or are to occur, the target device may automatically tilt thedashboard display and render images in perspective with lines of sightof the viewer when these images are rendered on the tilted dashboarddisplay.

Additionally, optionally or alternatively, perspective rendering asdescribed herein (e.g., with respect to a rendered portion of an imageon a tilted display screen, etc.) can be applied to rendering augmentedreality objects (e.g., objects that are superimposed onto images, etc.),text (e.g., closed caption texts overlaid with images, etc.),computer-generated graphics (e.g., advertisement messages, labels,posters, signs, etc.) overlaid with images, etc., in perspective withlines of sight of the viewer when these images are rendered with atarget device such as the tilted dashboard display in the presentexample.

7. Augmented Reality

FIG. 3D illustrates an example configuration in which an augmentedreality (AR) graphics superimposition unit 314 uses light map basedcontrol output such as 214 of FIG. 2 to perform AR graphicssuperimposition operations. As used herein, AR graphics may refer tographics that depict one or more AR objects or persons. An AR object orperson may be derived from other images other than the images to besuperimposed with. An AR object or person may also be derived fromcomputer generated graphics.

In some embodiments, the AR graphics superimposition unit (314)comprises software, hardware, a combination of software and hardware,etc., configured to receive image content 320, receive the light mapbased control output (214), receive AR graphics (e.g., 318, etc.) froman AR graphics input unit (306), perform the AR graphics superimpositionoperations that superimpose or overlay the AR graphics (318) with theimage content to generate AR superimposed image content, etc. In someembodiments, the AR graphics (318) is superimposed with the imagecontent based at least in part on the light map based control output(214), etc. In some embodiments, an AR object or person superimposedinto an image under techniques as described herein may becreated/generated not based on the same light sources as represented bythe light map based control output (214).

In some embodiments, the light map based control output (214) and theimage content (320) (e.g., collectively, in respective sub-bitstreams ofan overall media data bitstream, etc.) may be carried by and decodedfrom a media data bitstream. The light map based control output (214)may comprise a light source image captured/acquired contemporaneouslywhen an image of the image content (320) was originally created. The ARgraphics superimposition unit (314) may be configured todetermine/predict/estimate (e.g., using image based lighting techniquesbased on a spherical image or a light source image, etc.) lightconditions on an AR object of the AR graphics (318) as if the AR objectwere originally at a superimposed position at an original scene fromwhich the image to be superimposed with was captured and from which thelight source image was acquired/generated, and superimpose the AR objectrealistically (as if the AR object were originally at the superimposedposition at the scene in terms of highlight areas, mid-tone areas, darkareas, temporal spatial dependent colorimetric properties, etc.) intothe image based on the predicted/estimated light condition on the ARobject.

Additionally, optionally, or alternatively, the AR graphicssuperimposition unit (314) may operate with a display manager (e.g., 302of FIG. 3A or FIG. 3B, etc.) to render the AR superimposed image contenton a reference image rendering surface (e.g., 140 of FIG. 1A, etc.).

AR techniques as described herein can be applied to a variety ofdisplay-related applications such as interactive game applications,virtual reality applications, applications deployed with head updisplays, etc.

In an example, the image may be taken at the scene in Burbank, Calif.,whereas the AR object or person (e.g., an actor, etc.) may be a part ofimage taken at a place other than the scene. The light source imageacquired when the image was taken at the scene may indicate that thereis glowing lava near the location at which the AR object or person is tobe superimposed. Accordingly, the AR object or person may berealistically superimposed based on the light source image to indicatehighlights and colors caused by the AR object or person being near theglowing lava, as if the AR object or person was derived from a differentimage in which no glowing lava existed.

In another example, an AR object or person may be captured from imagesfrom a first location of a video conference and superimposedrealistically into images captured at a second location of the videoconference as if the AR object or person were realistically at thesecond location of the video conference.

In some embodiments, to save transmission bandwidth, a light sourceimage may be represented in a relatively compact form using non-pixeldata constructs (e.g., in a detailed or find representation, in anapproximate or coarse representation, etc.). For example, the lightsource image may be represented by a few, a few tens, etc., of vectorsthat represent light sources as detected from directional image sensordata (e.g., at the scene, etc.). In some embodiments, the light sourceimage may comprise a few tens of vector points, a few hundreds of bytes,up to 30 kB, etc., and may be carried as scene metadata in a media datastream with other data such as the image that was captured by the lightsources as represented by the light source image. In variousembodiments, light source images can be periodically (e.g., every 100milliseconds, etc.), constantly, sporadically, etc., generated at ascene. Some or all of these light source images generated at the scenemay be included as scene metadata in a media data stream that also isencoded with images taken at the scene. Additionally, optionally, oralternatively, a media data stream as described herein may betransmitted from an upstream device to a recipient device such as onthat includes the AR graphics superimposition unit (314) in any of avariety of ways (e.g., over-the-air, over the cable, internetdownloading, streaming from a media content server, etc.).

8. Ambient Light Control

FIG. 3E illustrates an example configuration in which an ambient lightcontrol unit 322 uses light map based control output such as 214 of FIG.2 to perform ambient light control operations in an image renderingenvironment in which images are rendered.

In some embodiments, the ambient light control unit (322) comprisessoftware, hardware, a combination of software and hardware, etc.,configured to receive image content 320, receive the light map basedcontrol output (214), perform the ambient light control operations thatcause setting up specific ambient light conditions in the imagerendering environment for rendering specific images in the receivedimage content (320) in the image rendering environment, etc.Additionally, optionally, or alternatively, the ambient light controlunit (322) may operate with a display manager (e.g., 302 of FIG. 3A orFIG. 3B, etc.) to render the image content on a reference imagerendering surface (e.g., 140 of FIG. 1A, etc.).

In some embodiments, the light map based control output (214) and theimage content (320) (e.g., collectively, in respective sub-bitstreams ofan overall media data bitstream, etc.) may be carried by and decodedfrom a media data bitstream.

In an example, the light map based control output (214) may comprise alight source image captured/acquired contemporaneously when an image ofthe image content (320) was color graded in a color grading environment.The ambient light control unit (322) may be configured to set up ambientlight conditions in the image rendering environment (e.g., a hometheater, etc.) using light source information in a light source image,etc. As a result, as the image is being rendered, the image is viewed inthe image rendering environment that simulates the ambient lightconditions of a viewing environment in which the image was color gradedin the color grading environment, etc. This, for example, allows theoriginal artistic intent in the color grading of the image in the colorgrading environment to be conveyed to the viewer with high fidelity to amuch greater extent.

Ambient light control as described herein may be applied to a variety ofimage rendering environments. For instance, ambient light control may beimplemented in an image rendering environment in an automobile, avehicle, a ship, an airplane, etc. Light sources (e.g., light emitterssuch as white or color LEDs installed in a cabin, in a ceiling, wall,etc., of an interior space, etc.) in the image rendering environment maybe automatically controllable based on ambient light control generatedunder techniques as described herein to generate ambient lightconditions specified in the light map based control output (214).

Ambient light control may also be implemented in an image renderingenvironment in which live content is being rendered. For instance,ambient light control may be implemented in an image renderingenvironment in a video conference. Light source information at anear-end of the video conference may be provided in light source images(e.g., in the light map based control output (214), etc.) from thenear-end to one or more far-ends. At the one or more far-ends, the lightsource information received from the near-end may be used to controlambient light conditions at the one or more far-ends. Similarly, lightsource information at a far-end of the video conference may be providedin light source images (e.g., in the light map based control output(214), etc.) from the far-end to a near-end. At the near-end, the lightsource information received from the far end may be used to controlambient light conditions at the near-end.

In some embodiments, the light map based control output (214) maycomprise optical properties of a display screen with which the colorgrading operations in the color grading environment were performed. Atarget device (e.g., an image rendering device, a dashboard display, ahead up display device, etc.) as described herein may comprise one ormore optical property control mechanisms that can be used in an imagerendering environment to simulate the optical properties (e.g., lighttransmissive properties, light reflective properties, light scatteringproperties, light diffusive properties, etc.) of the display screen inthe color grading environment. Examples of optical property controlmechanisms may include, but are not necessarily limited to, any of:optical polarizers, light modulation layers, optical films, etc.

In some embodiments, some or all of light sources in an image renderingenvironment may be controlled with interaction with a viewer/user ratherthan controlled automatically without user interaction. For example, inresponse to a target device (e.g., an image rendering device, etc.)receiving and/or the target device generating light source information,which light source information indicates that one or more specific lightsources in the image rendering environment should be adjusted inspecific ways or turned off, the target device may be configured togenerate user interface output to inform a user that the light sourceshould be adjusted in specific ways or turned off. Examples of userinterface output may include, but are not limited to only, any of: oneor more audible alerts (e.g., buzzes, etc.), haptic alerts (e.g.,vibrations, etc.), visual alerts (e.g., visual indicators rendered(e.g., near edges, graphic overlays, transparent overlays, etc.) on thereference image rendering surface (140), etc. In an example, specificadjustments or turning off of one or more specific light sources viauser interaction may be for the purpose of simulating target ambientlight conditions of a scene, a color grading environment, a far-end of avideo link, etc. In another example, specific adjustments or turning offof one or more specific light sources via user interaction may be forthe purpose of eliminating unfavorable ambient light conditions (e.g.,glare, sunlight, incoming headlights, etc.) in the image renderingenvironment. The target device may be configured to provide feedbacks tothe viewer/user whether the ambient light conditions match orapproximate target ambient light conditions within a tolerance rangeand/or whether the ambient light conditions are optimal for renderingimages with high fidelity.

9. Example Process Flows

FIG. 4 illustrates an example process flow according to an exampleembodiment of the present invention. In some example embodiments, one ormore computing devices or components may perform this process flow. Inblock 402, a target device, or a light map based controller (e.g., 100of FIG. 1A or FIG. 1B, etc.) therein, acquires directional image sensordata with one or more directional image sensors.

In block 404, the target device generates a light source image based onthe directional image sensor data.

In block 404, the target device causes one or more operations to beperformed for an image based at least in part on light sourceinformation in the light source image.

In an embodiment, the light source image is generated in real time asthe directional image sensor data is being collected; at least one ofthe one or more operations is performed in real time or in near realtime with generating the light source image based on the directionalimage sensor data.

In an embodiment, the one or more operations are performed with one ormore of: a mobile phone device, a video game device, a television, atablet computer, a heads up display (HUD) device, a dashboard displaydevice, a display device operating in conjunction with one or morecomputers, a display device in a vehicle, a movie studio system, a hometheater system, a color grading system, etc.

In an embodiment, the one or more operations comprise one or more of:display management operations to render the image in an image renderingenvironment, device positional operations to reposition an imagerendering surface of an image rendering device, augmented realityoperations to superimpose an object or person into the image, controloperations to control ambient light conditions in an image renderingenvironment, etc.

In an embodiment, the one or more directional image sensors comprisesone or more of: built-in cameras, external cameras, directional imagesensors disposed with an accessory to a computing device, non-cameradirectional image sensors, CMOS-based directional image sensors,CCD-based directional image sensors, nanotube-based directional imagesensors, LED-based image sensors, solar cell arrays, quantum dotphotoconductors, photodiodes, polymer-based image sensors, combinationsof two or more types of directional image sensors, etc.

In an embodiment, the light source image represents one of: low spatialresolution light source images, medium spatial resolution light sourceimages, high spatial resolution light source images, spherical images,non-spherical images, 3D images, etc.

In an embodiment, the light source image comprises one of: pixel data,non-pixel-data, etc.

In an embodiment, the light source image comprises one or more of: lightdirections, light color maps, light intensity maps, etc.

In an embodiment, the target device is further configured to compute,based at least in part on the light source image, one or more of:dominant light sources, dominant light directions, dominant lightillumination sizes, dominant light reflection sizes, light colors, lightintensities, a position of a viewer's head, etc.

In an embodiment, the target device is further configured to compute, inreference to a viewer, one or more directional light maps based at leastin part on the light source information.

In an embodiment, the directional light maps comprise a directionallight map for one of: a reference image rendering surface, a surfaceother than a reference image rendering surface, a spatial volume, etc.

In an embodiment, the directional light maps comprise a directionallight map that is computed based on one or more of: a location of theviewer, a reference surface, optical properties of a display screen, thelight source information, etc.

In an embodiment, the light source map is represented in a coordinatesystem that is one of: a relative coordinate system in relation to areference image rendering surface, a coordinate system in relation to auniversal coordinate system independent of a reference image renderingsurface, etc.

In an embodiment, the target device is further configured to relocatethe light source map in the coordinate system based on locational sensordata collected by one or more of: motion sensors, position sensors,orientation sensors, gyroscopes, electronic compasses, accelerometers,GPS modules, etc.

In an embodiment, the light source image is generated in an imagerendering environment; the target device is further configured toperform: determining, based at least in part on the light source image,ambient light conditions incident on an image rendering surface;estimating, based on the ambient light condition incident on the imagerendering surface in combination with reflection and scatteringproperties of the image rendering surface, a prediction of how theambient light conditions affect image qualities for images rendered onthe image rendering surface; based on the prediction, setting one ormore of black levels, contrast settings, white points or primary colorsdifferently in different spatial portions of the image rendering surfacein dependence on different ambient light conditions in the differentspatial portions of the image rendering surface; etc.

In an embodiment, the light source image is generated in an imagerendering environment; the target device is further configured toperform: determining, based at least in part on the light source image,first ambient light conditions on an image rendering surface and secondambient light conditions on a surface other than the image renderingsurface; determining that the second ambient light conditions on thesurface are better than the first ambient light conditions on the imagerendering surface; automatically repositioning a display screen of animage rendering device to the surface with the second ambient lightconditions; rendering one or more images on the display screen of theimage rendering device at the surface with the second ambient lightconditions; etc.

In an embodiment, the light source image is generated in an imagerendering environment; the target device is further configured toperform: determining, based at least in part on the light source image,first ambient light conditions on an image rendering surface and secondambient light conditions on a surface other than the image renderingsurface; determining that the second ambient light conditions on thesurface are better than the first ambient light conditions on the imagerendering surface; interacting with a viewer of an image renderingdevice to cause the viewer to reposition a display screen of the imagerendering device to the surface with the second ambient lightconditions; rendering one or more images on the display screen of theimage rendering device at the surface with the second ambient lightconditions; etc.

In an embodiment, the light source image is generated at a color gradingenvironment where an image is color graded; the one or more operationsare ambient light control operations performed in an image renderingenvironment based at least in part on the light source image, one ormore ambient light.

In an embodiment, the light source image is generated at a scene wherean image is captured; the target device is further configured toperform: receiving an image representation of an object or personindependent of the image; superimposing the image representation of theobject or person into the image; changing, based at least in part on thelight source image, one or more image portions of the imagerepresentation of the object or person as superimposed into the image toone or more of: highlights, mid-tones, or dark areas; etc.

In an embodiment, the target device is further configured to change,based at least in part on the light source image, one or more of whitepoints or primary colors for one or more image portions of the imagerepresentation of the object or person as superimposed into the image.

In an embodiment, the scene is at a specific location of a videoconference; the object or person represents a participant not at thespecific location of the video conference.

In an embodiment, the target device is further configured to perform:detecting incoming light in a viewer's viewing angle; moving an imagerendering surface away from a direction of the incoming light to adirection different from the direction of the incoming light; renderingone or more images on the image rendering surface in the directiondifferent from the direction of the incoming light; etc.

In an embodiment, the target device is further configured to perform:tilting a display screen in relation to a viewer's viewing angle;rendering at least a portion of an image on the tilted display screenthat looks perceptually same as if the portion of an image were renderedon the display screen not tilted in relation to the viewer's viewingangle; etc.

In various example embodiments, an apparatus, a system, an apparatus, orone or more other computing devices performs any or a part of theforegoing methods as described. In an embodiment, a non-transitorycomputer readable storage medium stores software instructions, whichwhen executed by one or more processors cause performance of a method asdescribed herein.

Note that, although separate embodiments are discussed herein, anycombination of embodiments and/or partial embodiments discussed hereinmay be combined to form further embodiments.

10. Implementation Mechanisms—Hardware Overview

According to one embodiment, the techniques described herein areimplemented by one or more special-purpose computing devices. Thespecial-purpose computing devices may be hard-wired to perform thetechniques, or may include digital electronic devices such as one ormore application-specific integrated circuits (ASICs) or fieldprogrammable gate arrays (FPGAs) that are persistently programmed toperform the techniques, or may include one or more general purposehardware processors programmed to perform the techniques pursuant toprogram instructions in firmware, memory, other storage, or acombination. Such special-purpose computing devices may also combinecustom hard-wired logic, ASICs, or FPGAs with custom programming toaccomplish the techniques. The special-purpose computing devices may bedesktop computer systems, portable computer systems, handheld devices,networking devices or any other device that incorporates hard-wiredand/or program logic to implement the techniques.

For example, FIG. 5 is a block diagram that illustrates a computersystem 500 upon which an example embodiment of the invention may beimplemented. Computer system 500 includes a bus 502 or othercommunication mechanism for communicating information, and a hardwareprocessor 504 coupled with bus 502 for processing information. Hardwareprocessor 504 may be, for example, a general purpose microprocessor.

Computer system 500 also includes a main memory 506, such as a randomaccess memory (RAM) or other dynamic storage device, coupled to bus 502for storing information and instructions to be executed by processor504. Main memory 506 also may be used for storing temporary variables orother intermediate information during execution of instructions to beexecuted by processor 504. Such instructions, when stored innon-transitory storage media accessible to processor 504, rendercomputer system 500 into a special-purpose machine that is customized toperform the operations specified in the instructions.

Computer system 500 further includes a read only memory (ROM) 508 orother static storage device coupled to bus 502 for storing staticinformation and instructions for processor 504.

A storage device 510, such as a magnetic disk or optical disk, isprovided and coupled to bus 502 for storing information andinstructions.

Computer system 500 may be coupled via bus 502 to a display 512, such asa liquid crystal display, for displaying information to a computer user.An input device 514, including alphanumeric and other keys, is coupledto bus 502 for communicating information and command selections toprocessor 504. Another type of user input device is cursor control 516,such as a mouse, a trackball, or cursor direction keys for communicatingdirection information and command selections to processor 504 and forcontrolling cursor movement on display 512. This input device typicallyhas two degrees of freedom in two axes, a first axis (e.g., x) and asecond axis (e.g., y), that allows the device to specify positions in aplane.

Computer system 500 may implement the techniques described herein usingcustomized hard-wired logic, one or more ASICs or FPGAs, firmware and/orprogram logic which in combination with the computer system causes orprograms computer system 500 to be a special-purpose machine. Accordingto one embodiment, the techniques herein are performed by computersystem 500 in response to processor 504 executing one or more sequencesof one or more instructions contained in main memory 506. Suchinstructions may be read into main memory 506 from another storagemedium, such as storage device 510. Execution of the sequences ofinstructions contained in main memory 506 causes processor 504 toperform the process steps described herein. In alternative embodiments,hard-wired circuitry may be used in place of or in combination withsoftware instructions.

The term “storage media” as used herein refers to any non-transitorymedia that store data and/or instructions that cause a machine tooperation in a specific fashion. Such storage media may comprisenon-volatile media and/or volatile media. Non-volatile media includes,for example, optical or magnetic disks, such as storage device 510.Volatile media includes dynamic memory, such as main memory 506. Commonforms of storage media include, for example, a floppy disk, a flexibledisk, hard disk, solid state drive, magnetic tape, or any other magneticdata storage medium, a CD-ROM, any other optical data storage medium,any physical medium with patterns of holes, a RAM, a PROM, and EPROM, aFLASH-EPROM, NVRAM, any other memory chip or cartridge.

Storage media is distinct from but may be used in conjunction withtransmission media. Transmission media participates in transferringinformation between storage media. For example, transmission mediaincludes coaxial cables, copper wire and fiber optics, including thewires that comprise bus 502. Transmission media can also take the formof acoustic or light waves, such as those generated during radio-waveand infra-red data communications.

Various forms of media may be involved in carrying one or more sequencesof one or more instructions to processor 504 for execution. For example,the instructions may initially be carried on a magnetic disk or solidstate drive of a remote computer. The remote computer can load theinstructions into its dynamic memory and send the instructions over atelephone line using a modem. A modem local to computer system 500 canreceive the data on the telephone line and use an infra-red transmitterto convert the data to an infra-red signal. An infra-red detector canreceive the data carried in the infra-red signal and appropriatecircuitry can place the data on bus 502. Bus 502 carries the data tomain memory 506, from which processor 504 retrieves and executes theinstructions. The instructions received by main memory 506 mayoptionally be stored on storage device 510 either before or afterexecution by processor 504.

Computer system 500 also includes a communication interface 518 coupledto bus 502. Communication interface 518 provides a two-way datacommunication coupling to a network link 520 that is connected to alocal network 522. For example, communication interface 518 may be anintegrated services digital network (ISDN) card, cable modem, satellitemodem, or a modem to provide a data communication connection to acorresponding type of telephone line. As another example, communicationinterface 518 may be a local area network (LAN) card to provide a datacommunication connection to a compatible LAN. Wireless links may also beimplemented. In any such implementation, communication interface 518sends and receives electrical, electromagnetic or optical signals thatcarry digital data streams representing various types of information.

Network link 520 typically provides data communication through one ormore networks to other data devices. For example, network link 520 mayprovide a connection through local network 522 to a host computer 524 orto data equipment operated by an Internet Service Provider (ISP) 526.ISP 526 in turn provides data communication services through the worldwide packet data communication network now commonly referred to as the“Internet” 528. Local network 522 and Internet 528 both use electrical,electromagnetic or optical signals that carry digital data streams. Thesignals through the various networks and the signals on network link 520and through communication interface 518, which carry the digital data toand from computer system 500, are example forms of transmission media.

Computer system 500 can send messages and receive data, includingprogram code, through the network(s), network link 520 and communicationinterface 518. In the Internet example, a server 530 might transmit arequested code for an application program through Internet 528, ISP 526,local network 522 and communication interface 518.

The received code may be executed by processor 504 as it is received,and/or stored in storage device 510, or other non-volatile storage forlater execution.

11. Equivalents, Extensions, Alternatives and Miscellaneous

In the foregoing specification, example embodiments of the inventionhave been described with reference to numerous specific details that mayvary from implementation to implementation. Thus, the sole and exclusiveindicator of what is the invention, and is intended by the applicants tobe the invention, is the set of claims that issue from this application,in the specific form in which such claims issue, including anysubsequent correction. Any definitions expressly set forth herein forterms contained in such claims shall govern the meaning of such terms asused in the claims. Hence, no limitation, element, property, feature,advantage or attribute that is not expressly recited in a claim shouldlimit the scope of such claim in any way. The specification and drawingsare, accordingly, to be regarded in an illustrative rather than arestrictive sense.

What is claimed is:
 1. A method, comprising: acquiring directional imagesensor data with one or more directional image sensors; generating alight source image based on the directional image sensor data; causingone or more operations to be performed for an image based at least inpart on light source information in the light source image; wherein themethod is performed by one or more computing devices; wherein the lightsource image is generated in an image rendering environment, and furthercomprising: determining, based at least in part on the light sourceimage, ambient light conditions incident on an image rendering surface;estimating, based on the ambient light condition incident on the imagerendering surface in combination with reflection and scatteringproperties of the image rendering surface, a prediction of how theambient light conditions affect image qualities for images rendered onthe image rendering surface; based on the prediction, setting one ormore of black levels, contrast settings, white points or primary colorsdifferently in different spatial portions of the image rendering surfacein dependence on different ambient light conditions in the differentspatial portions of the image rendering surface.
 2. A method,comprising: acquiring directional image sensor data with one or moredirectional image sensors; generating a light source image based on thedirectional image sensor data; causing one or more operations to beperformed for an image based at least in part on light sourceinformation in the light source image; wherein the method is performedby one or more computing devices; wherein the light source image isgenerated in an image rendering environment, and further comprising:determining, based at least in part on the light source image, lightconditions in at least one portion of a background or surround of animage rendering surface; estimating, based on the light condition in theat least one portion of the background or surround of the imagerendering surface, a prediction of how the light condition in the atleast one portion of the background or surround affect image qualitiesfor images rendered on the image rendering surface; based on theprediction, setting one or more of black levels, contrast settings,white points or primary colors in one or more spatial portions of theimage rendering surface in dependence on the light condition in the atleast one portion of the background or surround of the image renderingsurface.
 3. The method of claim 1, wherein the one or more directionalimage sensors are configured to capture or sense directional light andto generate directional image sensor data with a determined spatialresolution.
 4. The method of claim 1, wherein the one or moredirectional image sensors are located outside an image renderingsurface.
 5. The method of claim 1, wherein the light source image isgenerated in real time as the directional image sensor data is beingcollected, and wherein at least one of the one or more operations isperformed in real time or in near real time with generating the lightsource image based on the directional image sensor data.
 6. The methodof claim 1, wherein the one or more operations are performed with one ormore of: a mobile phone device, a video game device, a television, atablet computer, a heads up display (HUD) device, a dashboard displaydevice, a display device operating in conjunction with one or morecomputers, a display device in a vehicle, a movie studio system, a hometheater system or a color grading system.
 7. The method of claim 1,wherein the one or more operations comprise one or more of: displaymanagement operations to render the image in an image renderingenvironment, device positional operations to reposition an imagerendering surface of an image rendering device, augmented realityoperations to superimpose an object or person into the image, or controloperations to control ambient light conditions in an image renderingenvironment.
 8. The method of claim 1, wherein the one or moredirectional image sensors comprises one or more of: built-in cameras,external cameras, directional image sensors disposed with an accessoryto a computing device, non-camera directional image sensors, CMOS-baseddirectional image sensors, CCD-based directional image sensors,nanotube-based directional image sensors, LED-based image sensors, solarcell arrays, quantum dot photoconductors, photodiodes, polymer-basedimage sensors, or combinations of two or more types of directional imagesensors.
 9. The method of claim 1, wherein the light source imagerepresents one of: low spatial resolution light source images, mediumspatial resolution light source images, high spatial resolution lightsource images, spherical images, non-spherical images, or 3D images. 10.The method of claim 1, wherein the light source image comprises one of:pixel data or non-pixel-data.
 11. The method of claim 1, wherein thelight source image comprises one or more of: light directions, lightcolor maps, or light intensity maps.
 12. The method of claim 1, furthercomprising computing, based at least in part on the light source image,one or more of: dominant light sources, dominant light directions,dominant light illumination sizes, dominant light reflection sizes,light colors, light intensities, or a position of a viewer's head. 13.The method of claim 1, further comprising computing, in reference to aviewer, one or more directional light maps based at least in part on thelight source information.
 14. The method of claim 13, wherein thedirectional light maps comprise a directional light map for one of: areference image rendering surface, a surface other than a referenceimage rendering surface, or a spatial volume.
 15. The method of claim13, wherein the directional light maps comprise a directional light mapthat is computed based on one or more of: a location of the viewer, areference surface, optical properties of a display screen, or the lightsource information.
 16. The method of claim 1, wherein the light sourcemap is represented in a coordinate system that is one of: a relativecoordinate system in relation to a reference image rendering surface, ora coordinate system in relation to a universal coordinate systemindependent of a reference image rendering surface.
 17. The method ofclaim 1, further comprising repositioning or reorienting the lightsource map in the coordinate system based on locational sensor datacollected by one or more of: motion sensors, position sensors,orientation sensors, gyroscopes, electronic compasses, accelerometers,or global positioning system (GPS) modules.
 18. The method of claim 1,wherein the directional image sensor data is captured with differentexposure values—EV—and interleaved for capturing both bright and darkimage portions, thereby identifying directions and positions of lightsources.
 19. The method of claim 18, wherein the directional imagesensor data is captured every even times with a first EV suitable forcapturing bright image portions, and wherein the directional imagesensor data is captured every odd times with a second EV suitable forcapturing dark image portions.
 20. The method of claim 1, wherein thelight source image is generated in an image rendering environment, andfurther comprising: determining, based at least in part on the lightsource image, first ambient light conditions on an image renderingsurface and second ambient light conditions on a surface other than theimage rendering surface; determining that the second ambient lightconditions on the surface are better than the first ambient lightconditions on the image rendering surface; automatically repositioning adisplay screen of an image rendering device to the surface with thesecond ambient light conditions; rendering one or more images on thedisplay screen of the image rendering device at the surface with thesecond ambient light conditions.